Acquisition of Stand-up Behavior by a Real Robot using Hierarchical Reinforcement Learning

In this paper, we propose a hierarchical reinforcement learning architecture for a robot with large degrees of freedom. In order to enable learning in a practical numbers of trials, we introduce a low-dimensional representation of the state of the robot for higher-level planning. The upper level learns a discrete sequence of sub-goals in a low-dimensional state space for achieving the main goal of the task. The lower-level modules learn local trajectories in the original high-dimensional state space to achieve the sub-goal speci ed by the upper level. We applied the hierarchical architecture to a three-link, two-joint robot for a task of learning to stand up by trial and error. The upper-level learning was implemented by Q learning, while the lowerlevel learning was implemented by a continuous actor-critic method. The robot successfully learned to stand up within 750 trials in simulation and then in an additional 170 trials using real hardware.